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Abstract 

Objectives In 2013, the Chicago Police Department conducted a pilot of a predictive 
policing program designed to reduce gun violence. The program included development 
of a Strategic Subjects List (SSL) of people estimated to be at highest risk of gun 
violence who were then referred to local police commanders for a preventive interven¬ 
tion. The purpose of this study is to identify the impact of the pilot on individual- and 
city-level gun violence, and to test possible drivers of results. 

Methods The SSL consisted of 426 people estimated to be at highest risk of gun 
violence. We used ARIMA models to estimate impacts on city-level homicide trends, 
and propensity score matching to estimate the effects of being placed on the list on five 
measures related to gun violence. A mediation analysis and interviews with police 
leadership and COMPSTAT meeting observations help understand what is driving 
results. 

Results Individuals on the SSL are not more or less likely to become a victim of a 
homicide or shooting than the comparison group, and this is further supported by city- 
level analysis. The treated group is more likely to be arrested for a shooting. 
Conclusions It is not clear how the predictions should be used in the field. One 
potential reason why being placed on the list resulted in an increased chance of being 
arrested for a shooting is that some officers may have used the list as leads to closing 
shooting cases. The results provide for a discussion about the future of individual-based 
predictive policing programs. 
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Introduction 

The term “predictive policing” has garnered significant interest in the law enforcement 
community as a potential way to increase the likelihood of preventing crime before it 
occurs. While the term has various operational definitions, predictive policing is 
typically comprised of two elements: a prediction model that uses an algorithm to 
identify instances of increased crime risk, and an associated prevention strategy to 
mitigate and/or reduce those risks (Perry et al. 2013; Ridgeway 2013). With the 
progress of more advanced analytics (also commonly referred to as predictive analytics, 
machine learning, or data mining methods), statistical approaches are shown to make 
more accurate predictions than traditional crime analysis methods in the lab (Berk 
2011; Berk and Bleich 2013; Cohen et al. 2007; Gorr and Harries 2003). By leveraging 
advanced analytics, police departments may be able to more effectively identify future 
crime targets for preemptive intervention. However, there is little experimental evi¬ 
dence from the field demonstrating whether implementing an advanced analytics 
predictive model, along with a prevention strategy—“predictive policing”—^works to 
reduce crime, particularly compared to other policing practices in the field. 

The impact of both prediction and prevention, that is, a predictive policing program, 
needs to be tested because there are plenty of reasons to believe improvements in the 
accuracy of predictions alone may not result in a reduction in crime. First, the 
predictions may not be actionable because the location and/or time are not precise 
enough. For example, predictions may identify census blocks at increased risk of crime 
in the following week, but for most police departments, this period is too long to 
efficiently and effectively implement a strategy to prevent a crime. Second, since the 
baseline accuracy of predictions is still relatively low, small improvements can be made 
to appear as large percentage improvements, when they are rather insufficient to make a 
difference in the real world. For instance, a method or model may improve the 
prediction of homicide perpetrators in a city in a year from 1 out of 100,000 people 
to 6 out of 100,000—a 500 % improvement—^but using the average homicide levels in 
cities, the new approach will still fail to identify nearly 99.5 % of homicide perpetrators. 
Third, the crime prevention strategies may not work. So in our example, even if a 
prediction model identifies five more future homicide perpetrators than traditional 
analysis, the prevention program may not stop them from committing a homicide. 
Not to mention, they may not even receive the prevention program—the new method 
identified 6 out of 100,000, meaning 100,000 people would need to receive prevention 
to avoid 6 homicides. Finally, law enforcement may choose not to use the predictions 
(noncompliance). And these are only a few of the many reasons that an enhanced 
prediction model or method may not lead to crime reduction in the field, so there is still 
a need for research studies to better understand if and how predictive policing programs 
work in practice. 

There is some experimental evidence of the impact of predictive policing strategies 
on crime, albeit limited. Two peer-reviewed field experiments of explicitly formulated 
predictive policing programs suggest results are mixed. One study, co-authored by 
predictive policing software developers, compared the use of predictive policing 
software to identify micro-places at high risk of crime to crime analysts manually 
labeling high-risk micro-places; police then conducted additional patrols in the identi¬ 
fied places (Mohler et al. 2015). The study found that the predictive policing tool better 
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recognized future crime risk and reduced crime in comparison to the manual labeling 
by the crime analysts. But the other study, identifying blocks at high risk within an 
intelligence-led policing paradigm, did not result in a reduction of property crime 
compared to business-as-usual hot spots mapping and policing (Hunt et al. 2014). 
The authors concluded that the failure to identify an effect could be because the 
program did not in fact work (theory failure), low statistical power, and/or lack of 
program fidelity in some treatment units (implementation failure). The contradictory 
findings may be due to differences in study design, experimental control, prediction 
accuracy, prevention implementation, or a dozen other factors, which are discussed in 
more detail later. 

In order to better understand the effects of an individual-focused predictive policing 
program in the field, this study analyzes a pilot program implemented in Chicago in 
2013 aimed at reducing gun violence. The theory behind the program is not dissimilar 
to prior efforts to identify, monitor, and deter or incapacitate high-risk or highly active 
offenders to reduce crime (Abrahamse et al. 1991; Martin and Sherman 1986; 
Ridgeway et al. 2011; Sherman and Berk 1984). One difference, however, is that the 
individuals at high risk of being involved in crime in the future were identified using a 
predictive policing strategy based on a statistical model of co-arrest networks in a 
policing context. Importantly, the high-risk individuals were not necessarily under 
official criminal justice supervision nor were they identified through intelligence to 
be particularly criminally active. 

The predictive policing strategy examined in this study refers to the pilot, or first 
phase, of the Chicago Police Department’s (CPD) larger predictive policing program 
where individuals at highest risk for gun violence were placed on a Strategic Subjects 
List (SSL). The SSL was disseminated by central command and the prevention strategy 
was deferred to district commanders who decided the relevant policing intervention 
strategy for SSL individuals in their district. We test whether the introduction of the 
SSL affected city-level homicide rates. Furthermore, using individual-level data, we 
apply propensity score matching methods to estimate the impact of the SSL on the 
likelihood of high-risk individuals being involved in gun violence. Lastly, we test 
hypotheses for why the program may or may not have worked. 

The rest of this paper is structured as follows: in section two, we provide a review of 
previous literature on the prediction and prevention of criminal behavior at the indi¬ 
vidual-level, focusing on linking predictions to practice within the intelligence-led 
policing paradigm. Section three provides a description of Chicago’s SSL program 
and how it was implemented during the pilot period. In section four, we present the data 
and quasi-experimental methods used to evaluate program effectiveness and to identify 
the mechanisms driving outcomes. Section five presents results, and section six 
concludes with a discussion of what the findings mean for police departments looking 
to develop and implement similar programs. 


Literature review 

Predictive policing is a proactive policing model that was popularized, in part, by the 
development of advanced analytics that were lauded as highly successhil in other fields 
(Beck and McCue 2009; Berry and Linoff 2004; Zikopoulos and Eaton 2011). Now 
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that analysts have increasing methodological and computational sophistication to 
predict future crime patterns (Berk and Bleich 2013; Cohen et al. 2007; Perry et al. 
2013), many have claimed that police should be using these models to reorient their 
officers toward future, rather than current, problems (Beck and McCue 2009). The units 
of crime predictions can fall across a continuum of targets, ranging from large to small 
geographically sized areas (e.g., predict when and where crime is more likely to occur), 
all the way down to individuals (e.g., predict who is more likely to be involved in 
criminal activity). As the models differ in their predicted targets, we might expect these 
models to be tied to policing strategies for crime control purposes. The policing 
strategies would likely differ in terms of specificity of the prevention mechanism 
(e.g., from general prevention to targeted prevention of particular crime types or of 
specific behaviors) and of the time period (e.g., how far into the future the strategy 
might operate). However, the best ways to translate predictions into practice are still 
underdeveloped (Ridgeway 2013). 

Predictive policing comes from a long history of proactive policing strategies 
focused on getting ahead of crime before it occurs (Bordua and Reiss 1966). Since 
the 1980s, a large body of robust literature has grown on the effectiveness of proactive 
policing techniques for reducing crime (Braga et al. 2012; Mazerolle et al. 1998, 2000; 
Sherman and Weisburd 1995; Weisburd and Mazerolle 2000). The evidence-based 
policing field is replete with studies that demonstrate that the police can proactively 
prevent and reduce crime, and each share a commonality in the broadest sense with one 
another and predictive policing—a basis for the selection of a particular target and a 
mechanism to prevent and reduce crime. Multiple examples of these studies can be 
found on the Evidenced-Based Policing Matrix (Lum et al. 2011), and one can conceive 
of integrating predictive analytics into most of these programs. 

Predicting crime using individual-focused modeling 

To date, much of the interest in predictive policing has focused on using geospatial 
modeling to predict future hot spots (Beck and McCue 2009; Groff and La Vigne 2002; 
Hunt et al. 2014; Mohler et al. 2015; Perry et al. 2013). In fact, researchers started 
assessing computational methods for what they called “predictive crime mapping” over 
10 years ago (Gorr and Harries 2003; Groff and La Vigne 2002). Crime control 
programs that focus on geographic targets have been met with the most success 
(Braga 2005; Braga and Weisburd 2010; Sherman 1986; Sherman et al. 1997; 
Sherman and Weisburd 1995; Weisburd and Mazerolle 2000). Models and methods 
to predict the criminal behavior of individuals are less prominent in the criminal justice 
crime prevention literature. 

While more recent advanced analytic approaches have focused on improving 
geographic predictions, predicting future dangerousness of individuals is nothing 
new. For decades, researchers have been developing clinical predictions of future 
dangerousness using subjective approaches, such as intelligence or expert opinion 
assessments of “high risk” (Abrahamse et al. 1991; Bar-Hillel 1980; Martin and 
Sherman 1986; Pate et al. 1976) and static crime analysis (Braga et al. 2012; Eck 
et al. 2005; Ridgeway 2013; Sherman and Weisburd 1995). Subjective-based predic¬ 
tions have been replaced by more modem, reliable, and valid actuarial methods (Grove 
and Meehl 1996; Litwack 2001). Generally, these methods apply mathematical models 
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to administrative data in order to conduct risk assessments and predict future danger¬ 
ousness. Examples here include models predicting the risk of persons under community 
supervision reoffending (Berk 2008, 2011; Berk and Bleich 2013; Berk et al. 2009; 
Wright et al. 1984) and models assessing the risk that a gang affiliate will be involved 
in violence as a function of their social relationships (Papachristos 2009; Papachristos 
et al. 2011, 2012). Thus far, these models have been found to have a moderate level of 
predictive accuracy (Yang et al. 2010). 

These actuarial risk assessments have not been used in a policing context until now. 
Police traditionally use intelligence and subjective assessments to identify and monitor 
high-risk individuals, whereas actuarial risk assessments are relatively standard practice 
in other parts of the criminal system, such as for correctional placement, court sentenc¬ 
ing, and probation decisions (Dvoskin and Heilbmn 2001; Quinsey et al. 2000; Wright 
et al. 1984). These tools and methods are routinely integrated into decisions about 
supervision and sentencing, but researchers have warned that they were designed to 
faciltate efficient management of institutional resources, not to target individuals, and 
caution should be applied to ensure their proper use. The challenges of predicting future 
offending behavior and people’s misunderstanding of predictions have been well 
documented (Bar-Hillel 1980). For the models assesssing a person’s future risk of 
offending or victimization, a key complication is that, while the models can identify 
increased risk, the overall risk can still be very low. Indeed, a “very high risk” person 
for homicide might have a risk rate of 1 % per year. According to some scholars, this 
may still lead to cost-effective policing strategies, as the cost of a “false positive”, or 
someone who is incorrectly identified as a potential offender, is likely less costly than a 
“false negative,” or someone who was incorrectly identified as a non-offender (Berk 
2011). To further complicate the issue, researchers note that decisions using actuarial 
predictions may advance the continued marginalization of economically- and 
politically-disenfranchised populations (Silver and Miller 2002), which could be more 
detrimental in a policing context compared to someone in custody who has already 
been found guilty. 

Prevention: intelligence-led policing paradigm 

While the prevention strategies connected to any particular prediction may vary in 
specificity (e.g., general vs. targeted prevention) and the level of proactivity, the process 
of matching them is most closely described as an Intelligence-Led Policing (ILP) 
approach. A precise definition of Intelligence-Led Policing is difficult to nail down, 
but most broadly, it is a strategy that integrates data analysis and intelligence to help 
police prioritize their targets and activities (Ratcliffe 2002). Intelligence-led policing 
grew in popularity throughout the 1990s and early 2000s (Cope 2004), and according 
to Ratcliffe (2002, 2012), the defining characteristic is that intelligence is used as a 
“decision-making tool” to help the police prioritize their work effectively and efficient¬ 
ly to reduce crime, sometimes using external partnerships as a force multiplier. 
Intelligence-led policing can be situated at a variety of different organizational levels 
because “intelligence” can be used to inform programs, strategies, and even larger 
administrative priorities and policies. There are three components of an intelligence-led 
policing strategy, described in Ratcliffe’s “3i” model (Ratcliffe 2005): interpreting the 
available information (in predictive policing, through the use of a predictive model); 
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using the information to influence an agency’s decision-makers to adapt a crime- 
reduction strategy; and executing the strategy to impact criminal behavior and ideally, 
reduce crime (in predictive policing, through the use of a prevention program). 

Evidence on the effectiveness of a well-implemented ILP paradigm in general 
appears to be lacking in one way or another. ILP is a framework or model and not a 
singular program, which makes evaluation very challenging because it can be imple¬ 
mented very differently across different settings (treatment heterogeneity). Research 
tends to focus on organizational issues in adopting a tme ILP model at different levels 
(Ratcliffe 2002, 2005; Ratcliffe and Guidetti 2008). Ratcliffe (2012) reviewed the 
effectiveness of a few different programs in ILP frameworks with different methods, 
targets, and settings, and found that the frameworks met with some success. However, 
it may be that ILP alone is not enough to create success. It is likely the framework must 
be paired with evidenced-based programs to create positive results (Ratcliffe 2002). 
This is not unlike other disciplines, such as health (Chinman et al. 2004) and education 
(Sailor et al. 2008) in which evidence-based frameworks or models are built to include 
evidence-based programs that can vary over time and across locations. 

Connecting individual-focused predictions to practice 

ILP is a framework for applying interventions in which specific interventions must be 
developed separately. There are a number of interventions that can be directed at 
individual-focused predictions of gun crime because intervening with high-risk indi¬ 
viduals is not a new concept. There is research evidence that targeting individuals who 
are the most criminally active can result in significant reductions in crime (Braga and 
Weisburd 2012; Gendreau et al. 1996; Lipsey 1999; Loeber and Farrington 1998; 
Martin and Sherman 1986; Sherman et al. 1997). Additionally, successful programs 
that work with these high-risk offenders have also demonstrated cost-effectiveness 
(Caldwell et al. 2006; Foster and Jones 2006). For example, Martin and Sherman 
(1986) found that Washington, DC’s selective apprehension program did arrest repeat 
offenders more frequently than they would have been arrested otherwise, but the study 
did not examine whether the increased arrests reduced crime. 

Conversely, some research shows that interventions targeting individuals can 
sometimes backfire (McCord 2003; Sherman 1992; Welsh and Rocque 2014). As 
an example, some previous proactive interventions, including increased arrest of 
individuals perceived to be at high risk (selective apprehension) and longer 
incarceration periods (selective incapacitation), have led to negative social and 
economic unintended consequences. Auerhahn (1999) found that a selective 
incapacitation model (Greenwood and Abrahamse 1982) generated a large number 
of persons falsely predicted to be high-risk offenders, although it did reasonably 
well at identifying those who were low risk. At the extreme, “Three Strikes and 
You’re Out” laws were intended to incapacitate indefinitely those labeled as 
lifetime high-rate offenders (“predators”). Gottfredson and Hirschi (1986) found 
that such lifetime offender groups do not exist (i.e., propensity to engage in crime 
drops dramatically with age regardless of risk factors present earlier), and thus 
their incapacitation was unlikely to have prevented crime. Indeed, one study found 
three strikes was positively associated with homicides and had no statistically 
significant impact on crime rates (Kovandzic et al. 2004). 
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Models engaging small groups of criminally active offenders with policing actions 
have seen more recent interest. A meta-analysis of focused deterrence (also known as 
“pulling levers” models, e.g., Kennedy 1996 and McGarrell et al. 2006) found such 
interventions promising at reducing community crime and violence (Braga and 
Weisburd 2012). However, those selected for focused deterrence interventions were 
identified through manual police and community efforts, not through predictive ana¬ 
lytics. And, notably, the studies did not follow the individuals who were targeted, but 
instead examined their impact on community crime rates, with the exception of one 
evaluation that examined reduction of violence within targeted gang sets, but not 
necessarily program participants (Papachristos and Kirk 2015). 

Prevention strategies could affect the likelihood of criminal activity through four 
potential mechanisms: (1) treatment, which would change the internal motivation to 
offend (for high-risk people); (2) specific deterrence, which would change the external 
motivation to offend (for high-risk people); (3) incapacitation, which would limit the 
ability to offend (for high-risk people); and (4) general deterrence through perceiving 
more credible deterrence messaging and changing the social environment (for all 
people); see Fig. 1 for an illustration. 

Three of the four crime control mechanisms focus directly on the high-risk individ¬ 
ual, which according to research, is likely to be a much more effective way to target 
scarce resources because a small minority of offenders commit the majority of crimes 
(Blumstein 1986). Treatment works through changing the internal motivations for 
committing a crime and providing high-risk individuals with the skills they need to 
succeed (Gendreau et al. 1996; Sherman et al. 1997). Deterrence works through three 
factors associated with punishment: certainty, swiftness, and severity (Becker 1993; 
Cornish and Clarke 2014). People are assumed to act as if they are trading off the gains 
from crime with the costs of punishment and to weigh the present more heavily than the 
future (discounting). So when the expected benefit of committing a crime is greater 
than the expected costs, people commit crime. The greater the probability the crime 
prompts swift, certain, and severe punishment, for example, arrest or citation, the less 
likely someone will commit that crime, thus deterring the would-be criminal. The 
program may also reduce crime through an incapacitation effect whereby police may 
immediately increase incapacitation focusing on serving warrants or on violations of 
probation or parole for predicted individuals, which would remove their opportunity to 
commit crimes. Removing or reducing some of the criminally active population can 
even generate an overall “cooling” effect on others who were not targeted, thus leading 



Fig. 1 Crime control mechanisms for individual-focused predictive policing program 
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to further reduction in crime by those indirectly impacted by the crime control strategy 
(Braga and Weisburd 2012; Kennedy 1996). 


The Chicago SSL predictive policing program 
Overview 

This study investigates the predictive policing pilot program developed in collaboration 
between the Chicago Police Department (CPD) and the Illinois Institute of Technology 
(IIT), funded by the National Institute of Justice. ^ The aim of the pilot was to more 
efficiently and effectively target limited resources towards subjects in a community at 
risk for participation in gun violence, either as victims or perpetrators. IIT was 
responsible for the prediction model and used data contained within the CPD data 
warehouse to identify prior arrestees at a heightened risk for homicide (e.g., developed 
automated intelligence). The CPD led in the prevention strategy, using the computer¬ 
generated intelligence to action the intelligence. 

All five stages of ILP—(1) acquiring information, (2) analyzing intelligence/ infor¬ 
mation, (3) reviewing and prioritizing, (4) acting on intelligence and tasking responsi¬ 
ble parties with the plan, and (5) evaluating the impact of that action—^were included in 
the strategy. CPD worked on step 1 by upgrading their IT infrastmcture to allow them 
to automate the data collection from their data warehouse (Cope 2004). IIT worked on 
step 2, using the data to estimate an empirical model to generate data-driven intelli¬ 
gence. CPD worked on steps 3 and 4, with the Department of Operations first 
reviewing the list produced by IIT, and then tasking the district commanders to action 
the intelligence. While IIT continues to make model improvements and is currently on 
their fourth version (Lewin and Wemick 2015), this study evaluates implementation of 
model version 1.0. 

CPD prediction model (interpreting) 

The prediction model uses social networks (in the form of co-arrests) to previous 
homicide victims to predict the likelihood of someone becoming a victim of a homi¬ 
cide, and therefore essentially automating the intelligence production in the ILP 
strategy. The model focuses heavily on the recent body of literature examining corre¬ 
lations between victimization and the social connections to others who were victims of 
homicide (Papachristos 2009; Papachristos et al. 2011, 2012). The first model specifi¬ 
cation, version 1.0, estimates the relative risk of a subject being a homicide victim 
based on two input variables—the number of first-degree co-arrest links and the 
number of second-degree co-arrest links with previous homicide victims. A “first- 
degree link” refers to a relationship between a subject and an individual with whom the 
subject was previously co-arrested who later became a homicide victim. A “second- 
degree link” refers to a relationship in which a subject was co-arrested with another 
person who, in turn, was co-arrested with a later homicide victim (see Fig. 2). 


^ This is only one component of the larger long-term collaboration which broadly explores whether and how 
crime can be predicted. 
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Fig. 2 First- and second-degree co-arrest links 

The counts of co-arrest links were summed over the past 5 years and weighted for 
recency. A quadratic model was fitted to the data, meaning the probability of being a 
homicide victim increased at an increasing rate with respect to the count of links. Early 
model tests found that for some subjects (those with many links), the odds of being a 
homicide victim were thousands of times the risk of the general public over a 2-year 
follow-up window. To address the numerical instability^ reflected by these results, the 
maximum assigned risk multiplier was “500+”. This study does not evaluate the 
validity or reliability of the model, but rather focuses on the impact of the predictions 
being used in practice. 

CPD’s prevention strategy (influencing and impacting) 

The list of subjects and their risk scores were then forwarded to the CPD 
Deployment Operations Center (DOC). The DOC did a subjective review of the 
list, assigning subjects from different police districts to the SSL based on their 
scores and human intelligence collected by CPD. Each district was to be 
assigned a list of the 20 people in their district with the top risk scores in 
addition to all subjects with risk scores of “500+”. A total of 426 of some of 
the highest-risk individuals^ were put onto the SSL version 1.0 on March 26, 


2013. 


The individuals on the SSL were considered to be “persons of interest” to the CPD. 
District commanders were responsible for directing their personnel to act on this 
intelligence and were to be held accountable at regular COMPSTAT meetings. Com¬ 
manders were not given specific guidance on what treatments to apply to their SSL 
members; instead, they were expected to tailor interventions appropriately. The main 
guidance provided by central leadership was to use the programs detailed in Chicago’s 
Gang Reduction Strategies (Chicago Police Department 2014) on SSL members when 
possible,^ but commanders were left wide discretion as to what actions their units 
should take. 


^ Here, “numerical instability” refers to the likelihood that the estimates that an SSL member’s calculated risk 
of a specified thousands of times more likely to be killed is due more to statistical artifacts from fitting the 
quadratic curve than an accurate estimate. 

^ Initially, CPD said they would put all the highest-risk individuals on the list; however, they decided to vet the 
list through their Deployment Operations Center (DOC), who made some changes to who would appear on the 
list, and therefore, the 426 individuals did not represent the highest scoring individuals based on the model. 
"^Details of GVRS are available at http://directives.chicagopolice.org/directives/data/a7a57bf0-136dld31- 
16513 -6dl d-3 82b311 ddf65fd3a.html 
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Since the prevention strategy was decentralized to district commanders, we collected 
qualitative evidence to identify the prevention and intervention strategies used in the 
field. Specifically, from October 2013 to June 2014, the research team conducted 
interviews with district commanders (or their executive officers) and observed 
COMPSTAT meetings. A total of 43 semi-stmctured interviews were conducted across 
all 22 police districts. Commanders or executive officers from each district were 
interviewed at least once and a maximum of three times over the course of 9 months. 
Interviews lasted between 10 and 30 minutes and consisted of questions regarding the 
quality of the SSL, uses of the SSL in practice, and suggestions to improve the SSL. 
The research team also observed 48 COMPSTAT presentations by both area and 
individual district commanders across 17 COMPSTAT meetings. Structured data 
collection protocols were completed for both the interviews and the COMPSTAT 
observations, which were then entered into a spreadsheet. The detailed notes were then 
coded and content analyzed. 

The COMPSTAT observations demonstrate most districts did not focus on 
intervening with SSL subjects. In over two-thirds (68.8 %) of presentations 
observed, there was no mention of the SSL. In another 12.5 % presentations, 
there was an acknowledgment of the SSL, with no further discussion. In less than 
one in five (18.7 %) presentations, there was both a discussion and executive 
guidance, which consisted of: (1) allow beat officers to take the lead in contacting 
SSL subjects, (2) consider using fugitive location and district intelligence teams to 
locate SSL subjects, and/or (3) change the focus from arresting SSL subjects for 
minor offenses (for which they would be immediately released) to finding ways to 
detain SSL subjects over the long term. There was no evidence of executive 
follow-up on these recommendations at the meetings. 

Interview findings indicate two common themes for how commanders recom¬ 
mended addressing SSL subjects to beat officers during the pilot. First, officers (or 
teams of officers) were assigned to make contact with SSL subjects on varying 
schedules, usually by going to their home addresses or other locations (7 of 22 
districts, or 31.6 %). Second, officers were provided information about the iden¬ 
tities of the SSL subjects, and the officers were to make contact with the SSL 
subjects “if noticed”, especially if the subjects were acting suspiciously (10 
districts, 45.4 %). The remaining cluster of districts (5 districts, 22.6 %) reported 
some combination of both approaches. Interviews indicated that directing officers 
to increase contact with SSLs was likely the extent of the preventive intervention 
strategy for the majority of districts. In interviews with CPD staff, it was noted 
that the central command encouraged the local commanders to take advantage of 
enhanced legal sanctions authorized through the Gang Violence Reduction Strat¬ 
egy (Chicago Police Department 2014), but officers reported using these programs 
for a very small subset of SSL subjects. 

In sum, findings from the observations of COMPSTAT meetings and interviews 
with district-level administration suggest the topic of SSL subjects received relatively 
little attention. Overall, the observations and interview respondents indicate there was 
no practical direction about what to do with individuals on the SSL, little executive or 
administrative attention paid to the pilot, and little to no follow-up with district 
commanders. These findings led the research team to question whether this should be 
considered a prevention strategy. 
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Methods 

This study uses an intent-to-treat analysis to estimate the impact of the CPD SSL 
predictive policing program version 1.0 on city-wide homicide levels and on incidence 
rates of individual-level involvement with gun crime. Outcomes of the pilot are 
analyzed using two quasi-experimental methods. First, we perform an intermpted 
time-series analysis of city-level data to determine whether the introduction of the 
SSL changed the aggregate homicide crime trend. Second, we conduct propensity score 
analysis using individual-level data in order to determine whether being on the SSL 
affected the likelihood of being involved with gun violence. To do so, we exploit the 
fact that some of the subjects on the SSL identified as being at highest risk were not 
treated (i.e., their score met the criteria for inclusion on the SSL but they were excluded 
for reasons we describe later). Last, we test hypotheses regarding the mechanisms 
driving the effects. 

Data 

For the city-level analysis, this study used publically available data on CPD’s website 
(gis.chicagopolice.org). The outcome was monthly homicides from January 2004 
through April 2014. Examining a 10.75-year period, there was an average of 38 
monthly homicides (SD = 10.65). Homicide counts followed a distinct seasonal pattern 
and a general linear trend downward over the entire period. 

For the individual-level analysis, we used two datasets provided by CPD that cannot be 
made public due to the sensitive nature of the data (for aggregated descriptive statistics, see 
Table 1). The first dataset was a person-level file of 873,281 individuals with arrest 
histories prior to March 2013, including data on (1) demographics (gender, age at most 
proximate arrest, race), (2) arrest history (number and type), (3) social network variables 
(number of first- and second-degree co-arrestees who were victims of homicide), and (4) 
the risk score generated by IIT. This file also contained four outcomes of interest for the 1- 
year follow-up period: (1) murder victim, (2) shooting victim, (3) arrest for murder, and (4) 
arrest for a shooting. The second dataset contained all recorded police contact with the 
17,754 arrestees with at least one first- or second-degree association with a homicide 
victim and law enforcement from 1980 through the end of the observation window. This 
file contained 542,636 records, with 464,006 dating before the intervention period. 

Analysis 

City-level outcomes: interrupted time-series design (ARIMA) 

This paper uses a time-series analysis to better understand whether the introduction of the 
SSL version 1.0 affected the homicide trend in Chicago. Specifically, we tested whether it 
led to a reduction of gun violence at an aggregate level. The ARIMA models were based 
on two components: (1) a model of homicides as a time series, which makes inferences 
about the underlying process in the dependent outcome using different time-series 
variational components, and (2) an impact assessment of the introduction of the SSL 
version 1.0 (McCleary et al. 1980). The first step was to identify the model for the time 
series, which characterizes autoregressive, non-seasonal differences, and lagged forecast 
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Table 1 Sample descriptive statistics and balance table 


Variables 

SSLs 
n = 426 

Unweighted 
comp group 
n= 17,754 

Weighted 

comparison 

group 

ESS = 273 

Std. effect K Smirnoff p 
size statistic 


Mean SD 
or % 

Mean SD 
or % 

Mean SD 
or% 



Demographics 


Male 

African-American 

Age at last 

95.8 % 

77% 

22.57 

4.76 

90.4 % 

77.5 % 

26.23 

9.02 

96.9 % 

76.9 % 

22.91 

5.05 

-0.06 

0.00 

-0.70 

.01 

0.00 

.05 

.29 

1 

.74 

arrest 

Prior arrests 

Murder 

.05 

.25 

.03 

.19 

.06 

.24 

-0.15 

.01 

1 

Sexual assault 

.02 

.15 

.02 

.16 

.02 

.13 

0.02 

0 

1 

Robbeiy 

.49 

.91 

.37 

.77 

.53 

.88 

-.04 

.03 

1 

Aggravated 

.25 

.57 

.17 

.45 

.32 

.65 

-.12 

.04 

1 

batteiy 

Burglary 

.35 

.81 

.31 

.81 

.48 

.97 

-.17 

.06 

.51 

Theft 

.73 

2.08 

.77 

2.81 

.87 

2.19 

-.07 

.02 

1 

MVT 

.77 

1.24 

.61 

1.16 

.75 

1.18 

.02 

.02 

1 

Arrest 

3.76 

4.62 

3.60 

4.51 

3.98 

4.82 

-.04 

.03 

1 

Outstanding warrant 

1.21 

1.73 

.88 

1.47 

1.18 

1.75 

.02 

.04 

.88 

Prior contact with CPD 

All arrests 

18.87 

12.19 

11.28 

9.51 

18.13 

12.74 

.07 

.06 

1.00 

Contact cards 

40.83 

84.12 

12.18 

20.18 

34.46 

58.55 

.08 

.07 

.40 

Victimizations 

.85 

1.18 

0.82 

2.13 

.95 

1.56 

-.08 

.03 

.31 

Risk 

# 1st Degree 

1.16 

1.08 

.24 

.46 

1.04 

.92 

.12 

.07 

.42 

# 2nd Degree 

7.28 

5.70 

1.46 

1.40 

6.25 

5.31 

.18 

.12 

.01 

Risk score 

268.58 

195.15 

30.48 

34.61 

208.62 

177.14 

.31 

.16 

<.01 


errors, and the second step was to model in intervention effects (Bruinsma and Weisburd 
2014). Once the appropriate model was selected, we also conducted a series of sensitivity 
analyses to test if there were other breaks in the time series not related to SSL.^ 

This study uses the ARIMAX R procedure (Ohri 2013) to test the impact of the SSL 
on overall city-level homicides in Chicago. ARIMAX identified the best fitting model 
of the entire time trend as ARIMA (0,0,0)(2,0,1), meaning the homicide series does not 
display non-seasonal variation, yet it is characterized by the following seasonal 


^ It is important to note that Chicago has gone through transformative changes over this time period, including 
a new Superintendent in 2011 and the integration of COMPS TAT to provide oversight. In addition to changes 
in leadership and management style, CPD has implemented a large number of homicide reduction strategies 
during this time period, including the multiple changes to the Gang Violence Reduction Strategy, and gang 
call-ins across different districts starting in 2010. 
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processes: two seasonal autoregressive terms, no differences across seasons, and one 
seasonal moving average term. 

Individual level outcomes: propensity score matching design 


Chicago PD implemented the program citywide and was not willing to randomize high- 
risk subjects to the SSL, so an experimental setting for evaluating program effectiveness 
on the individual-level was not achieved. CPD originally stated that they would treat the 
top 20 individuals in each district, and everyone who scored above a certain threshold 
(500+), which would lend itself to statistical approaches for quasi-experimental settings 
such as a regression discontinuity design or instmmental variable approach. However, 
there was, in fact, quite a substantial overlap in risk scores between the SSL and a group of 
individuals that did not end up on the list, who became our pool of non-treated potential 
comparison cases (see Fig. 3). This overlap of treated (i.e., on the SSL) and untreated (i.e., 
not on the SSL) happened for two reasons: (1) some districts did not have a large number 
of the highest-risk people in their area of operation, so individuals with slightly lower 
scores appeared on their list; and (2) the DOC had some discretion on who to put on the 
SSL, particularly when there were a lot of very high-risk individuals, so not all of the 
highest scoring individuals were ultimately placed on the SSL. This allowed us to match 
risk scores, along with other observed features that research indicates are associated with 
predicting hiture criminal offending, in order to generate a weighted comparison group 
that was almost statistically indistinguishable from the SSL on all observable measures.^ 
Individuals were selected into treatment by their risk scores and, therefore, they 
differed systematically from eligible comparison cases. In order to control for these 
differences, case weights, w^, were estimated to alter the covariate distribution (includ¬ 
ing the risk scores) for the comparison group. This results in equivalent comparison and 
treatment groups (to create treatment effects on the treated estimates, as opposed to 
average treatment effects). These weights were created using boosted regression to 
provide the conditional odds of receiving treatment where is the vector of control 
variables and p{xi) is the estimated conditional probability of receiving treatment for an 
individual with control variables equal to x^, also known as the propensity score 
(McCaffrey et al. 2004; Rosenbaum and Rubin 1983): 


Wi = 


p{xi) 

\-p{Xi) ■ 


Using the weighting scores generated in the TWANG R package (Ridgeway et al. 
2014), the following equation reduces the bias of estimates of the effective average 
treatment effect on the treated (EATT): 


EATT = 


n 5 


^ While there is always the possibility that the groups are different on unobservable variables, we have 
captured many of the important research-validated criminogenic factors. That is why we specify the approach 
reduces, rather than eliminates, bias. 
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■ Comp 

■ SSL 


Fig. 3 Overlap of risk scores between SSLs and comparison group 



where y is the outcome variable and t is the treatment indicator, for all individuals i. 
After applying the propensity score weights, the comparison group was reduced from 
17,754 unweighted cases to an effective sample size of 273, and the only significantly 
different pre-treatment covariates were the number of second-degree associates and the 
risk scores. Since these two variables were still unbalanced (e.g., the treatment group 
still had higher scores than the weighted comparison), we used these variables as 
covariates in all weighted regressions and estimated doubly-robust models (Funk 
et al. 2011; Huber 1973). Doubly-robust models add remaining unbalanced variables 
as predictors to control for any residual between-group differences to create the closest 
statistical match between treatment and control groups as possible. According to 
scholars, this is a valid way of controlling for unbalanced and missing variables for 
causal modeling with propensity score matching (Kang and Schafer 2007). 

Estimates of the impact of the predictive policing strategy could be biased if there 
were other programs targeting our control and/or treatment groups. While there were 
many violence reduction initiatives taking place in Chicago, including call-ins, almost 
none of the SSL subjects were included. An exception was one district (out of 22), 
which was a pilot area of the program that issued focused deterrence notification letters 
to SSL subjects. The commander (or her designee), along with a representative from a 
non-profit that coordinates services to ex-offenders, visited the SSL subjects or their 
families to let the subject know that he or she was identified at being at heightened risk 
for homicide victimization. This was the only place where a formal “treatment” was 
offered as a way to prevent gun crime, but we had no way to track if any SSL subject 
received services. As only one district participated, less than 5 % the SSLs were subject 
to this intervention, so we would not expect this to be driving our results. 

Mediation analysis 

In an effort to better understand what may be driving an association between inclusion 
on the SSL and our individual-level outcomes (involvement in gun violence), we 
conducted a mediation analysis. The mediation analysis allows us to explain the 
mechanisms that underlie an observed relationship between the SSL and involvement 
in the commission of a gun crime via the contacts with police. That is, we hypothesize 
the program was designed to enhance the deterrence message (through the probability 
of getting caught) not only directly but also by delivering prevention strategies that 
could affect decision-making that would lead to a weapons offense (as either an 
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offender or a victim). It may also have an incapacitation effect by removing the SSLs 
from the community at a higher rate. In this analysis, we try to tease out whether the 
individual-level effects of being on the SSL are based on deterrence or incapacitation. 

Since the prevention strategy most consistently described for every district of 
Chicago was contact with SSL members, we considered contact with police as a 
mediator variable by first testing whether those on the SSL experienced greater contact 
with police than the comparison group. We then conducted a mediation analysis in 
which the likelihood of involvement with a gun crime as either a victim or offender 
depends on being on the SSL (direct effect) and on the extent of contact with police, 
which is a function of being on the SSL (mediation effect). 


Results 

Impact 

City-level homicide rates 

Examining the raw data, it is clear that homicide trends display a high degree of 
seasonality, with more homicides occurring in the warm months.^ There is a negative 
linear trend across the series (b = -0.0267), shown as the dotted line in Fig. 4, with 
overall monthly homicides decreasing from January 2004 through September 2014. 
Visually, it appears the homicide trend was falling prior to the introduction of the SSL 
in April 2013; the SSL version 1.0 was released to the district commanders at the end of 
March. The ARIMA analysis was conducted to statistically test whether the introduc¬ 
tion of the SSL affected the monthly homicide levels in Chicago. 

When we entered the month that the SSL was introduced to the ARIMA (0,0,0)(2,0,1) 
model in April, we found that there was a decrease in monthly homicides of3.90 (95 % Cl: 
0.22, 7.58). However, sensitivity analyses examining decreases in homicide that pre- and 
post-date the SSL program show that this decrease is likely to be part of an overall trend 
downward, and not specifically due to the SSL intervention. Specifically, we conducted a 
set of “placebo tests”, where we modeled in a fake intervention date and tested whether 
there was still an effect. The analysis indicated a statistically significant reduction in monthly 
homicides when using 7 months pre- and post-SSL introduction. This demonstrates there 
was a statistically significant reduction in monthly homicides each month above and beyond 
the longer-term linear trend prior to the intervention (Table 2). With this sensitivity analysis 
in mind, we conclude that the statistically significant reduction in monthly homicides 
predated the introduction of the SSL, and that the SSL did not cause fiirther reduction in 
the average number of monthly homicides above and beyond the pre-existing trend. 

Impact on individual risk of homicide 

Between March 2013 and March 2014, there were 405 homicides in Chicago. Seventy- 
nine percent of the homicide victims had a criminal history in the years prior to their 
deaths, 16 % had at least one association with a homicide victim, and 1 % of them were 


^ With the exception of the winter of 2012 which did not experience the same degree of a “cooling off’ period. 
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Fig. 4 Monthly homicides in Chicago from January 2004 to September 2014 

listed on the SSL (see Fig. 5). Looking at this from another perspective, 0.7 % of the 
426 SSL subjects were homicide victims, 0.4 % of the 17,754 associates were homicide 
victims, 0.029 % of the 855,527 former arrestees with no associates were homicide 
victims, and 0.003 % of the rest of the almost 2 million Chicago residents without any 
criminal record were victims of homicide. 

A greater proportion of individuals on the SSL were involved in a shooting as either 
a victim or an arrestee, 6.^ %(n = 29), than comparable former arrestees who were not 
placed on the list but had at least one first- or second-degree association with a 
homicide victim (3.2 %, ^ = 529), and 0.2 % of former arrestees with no linkages to 
prior homicide victims (n= 1,939). However, once other demographics, criminal his¬ 
tory variables, and social network risk have been controlled for using propensity score 
weighting and doubly-robust regression modeling, being on the SSL did not signifi¬ 
cantly reduce the likelihood of being a murder or shooting victim, or being arrested for 
murder. Results indicate those placed on the SSL were 2.88 times more likely to be 
arrested for a shooting (Table 3). 

Mediation analysis 

Seventy-seven percent of the SSL subjects had at least one contact card over the year 
following the intervention, with a mean of 8.6 contact cards, and 60 % were arrested at 
some point, with a mean of 1.53 arrests. In fact, almost 90 % had some sort of interaction 


Table 2 Placebo sensitivity analyses: all homicides 


Intervention 

month 

Intervention 

coefficient 

SE 

Oct 2012 

-4.23* 

2.15 

Nov 2012 

-4.13* 

2.18 

Dec 2012 

-4.52** 

1.39 

Jan 2013 

-4.46** 

1.68 

Feb 2013 

-6.33** 

1.89 

March 2013 

-5.54** 

1.93 

April 2013 

-3.90** 

1.88 

May 2013 

-3.00 

2.36 


V<0.05, **/7<0.01 
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■ No CJ Record ■ No Associates ■ 1 + Associates ■ SSL 


N=3, 1% 



N=253, 62% 


Fig. 5 Homicides in Chicago from March 2013 to March 2014 by risk group 


with the Chicago PD (mean= 10.72 interactions) during the year-long observation win¬ 
dow. This increased surveillance does appear to be caused by being placed on the SSL. 
Individuals on SSL were 50 % more likely to have at least one contact card and 39 % more 
likely to have any interaction (including arrests, contact cards, victimizations, court 
appearances, etc.) with the Chicago PD than their matched comparisons in the year 
following the intervention. There was no statistically significant difference in their 
probability of being arrested or incapacitated^ (see Table 4). One possibility for this result, 
however, is that, given the emphasis by commanders to make contact with this group, 
these differences are due to increased reporting of contact cards for SSL subjects. 

Results of the impact analysis show the only statistically significant difference 
between the comparison and SSL group was an increase in arrest for shooting. 
Therefore, we analyzed whether the impact of SSL membership on being arrested for 
a shooting was mediated by additional contact. Results indicate that the total relation¬ 
ship between treatment and the outcome is 1.18, and most of that comes from the direct 
effect—^not mediation (Fig. 6). In other words, the additional contact with police did 
not result in an increased likelihood for arrests for shooting, that is, the list was not a 
catalyst for arresting people for shootings. Rather, individuals on the list were people 
more likely to be arrested for a shooting regardless of the increased contact. 


Discussion 

Several important findings have emerged from the current pilot. First, while using 
arrestee social networks improved the identification of future homicide victims, the 
number was still too low in the pilot to make a meaningful impact on crime. The pilot 
version 1.0 of the model identified less than 1 % of homicide victims (3 out of 405) so 
there is certainly room for improvement. Second, the prevention strategy associated 
with the predictive strategy was not well developed and only led to increased contact 
with a group of people already in relatively frequent contact with police. As such, the 
main result of this study is that at-risk individuals were not more or less likely to 
become victims of a homicide or shooting as a result of the SSL, and this is further 
supported by city-level analysis finding no effect on the city homicide trend. 


^ Most arrestees were not incapacitated for any significant period of time, but rather were booked into the 
Cook County jail and released within a few hours to a few days. 
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Table 3 Doubly robust treatment estimates 


Estimate (SD) 

t test 

Exp(b) 

Shooting victim 

-.221 (.397) 

-.558,/7 = .58 

.802 

Shooting arrest 

1.06 (.430) 

2.46, =.01* 

2.88 

Murder victim 

.037 (.669) 

.055,/7=.96 

1.04 

Murder arrest 

.453 (.462) 

.981,;7=.33 

1.57 

Any weapon outcome 

.168 (.300) 

.559,/?=.58 

1.18 


We do find, however, that SSL subjects were more likely to be arrested for a 
shooting. The effect size was rather large, such that those placed on the SSL were 
2.88 times more likely than their matched counterparts to be arrested for a shooting, 
although this is based on of a small absolute number of shootings—only 9 individuals 
from the SSL were arrested for a shooting in the year after being placed on the list, 
against 5 from the matched control group (or 84 from the unmatched comparison 
group). This raises some questions as to whether this is a “positive” or “negative” 
finding—did the program lead to an increase in shootings perpetrated by SSL subjects 
due to some sort of backfire or harmful effect, similar to one that has been identified in 
prior research (McCord 2003; Sherman 1992; Welsh and Rocque 2014), or were they 
more likely to be arrested for a shooting they perpetrated because they were under more 
intense surveillance and intelligence-gathering activities? 

We explore this question by examining the date of the shooting associated with the 
arrests, hypothesizing that a backfire effect could not occur before the introduction of 
the SSL. Unfortunately, the data do not lead to much clarification on this point because 
of missingness—^there was a shooting date associated with the arrest in only 56 % 
(^ = 5) of the SSLs (total n = 9) and 81 % (^ = 68) of the unweighted comparison group 
(total n = 84). In the SSL group, 80 % of the shootings occurred after the intervention 
date compared to the 88 % of the eligible unweighted comparison cases. Therefore, 
based on these available data, there does not appear to be a difference in the timing of 
the shootings that resulted in arrests during our observation window between SSL 
members and to the unweighted comparison group (see Table 5). 

While missing data make it impossible to confirm if the shootings associated with 
arrests occurred before or after treatment for 44 % of the SSL subjects, there are some 
reasons to believe the program did not increase shootings. If the existence of the SSL 
caused a backfire of shootings that would not have happened otherwise, committed by 
a group of people that the Chicago PD were more closely monitoring (and had 
significantly more contact with), we would have expected more mentions of the SSL 


Table 4 Doubly-robust difference in contact with CPD between SSL subjects and their matched comparisons 



Estimate (SD) 

t test 

Exp(b) 

Contact cards 

.393 (.147) 

2.68, p = .007 

1.48 

Arrest 

.195 (.145) 

1.34,p = .18 

1.22 

Any interaction 

.332 (.134) 

2.48, p = .01 

1.39 
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Fig. 6 Mediation analysis results 

subjects during interviews and COMPSTAT meetings. We found no evidence such a 
surge occurred, either in the media or in our interviews, and interviewees were typically 
forthright about problems with the pilot. 

Second, there is a large overlap between victims and offenders, and we find a 
program impact only on arrest for shootings and on none of the other indicators. 
Indeed, the risk of an SSL subject being a victim was lower than that of the control 
group, just not to a statistically significant level. Further evidence disputing the backfire 
theory is that we find no indirect relationship between increased police contact on being 
arrested for a shooting. Rather, it appears the list had a direct effect of increasing the 
probability of arrest, which did not work through police contact. We offer one expla¬ 
nation of this provided to us by the Chicago PD. They emphasize that the list was used 
as an intelligence-gathering source. Meaning, when there was a shooting, the police 
looked at the members of the SSL as possible suspects. This suggests that the impact of 
the SSL was on clearing shootings, but not on gun violence in general, during the 
observation window. However, a key question is why this did not lead to a reduction in 
the perpetration of gun violence. 

A further discussion of drivers of the null, city-level result is also warranted. The 
effect on the city-wide homicide trend is difficult to detect because homicide was 
trending downward before the introduction of the SSL. In the year after the introduction 
of the SSL, a greater proportion of the 426 SSL individuals were victims of homicides 
(0.7 %) than the 17,754 individuals who were previously known to law enforcement 
but were not placed on the list (0.036 %) or the over 2 million Chicago residents 
without criminal records (0.003 %). However, controlling for demographics, criminal 
history, and co-arrest social network characteristics, those on the SSL were no less 
likely to be the victims of a homicide or a shooting than those who were not placed on 
the list. Again, it is not clear if this is due to the absence of a defined prevention strategy 
or a lack of impact because this sort of approach cannot work (e.g., theory failure vs. 
implementation failure). 

The prediction model in this study was a first version model based on a simple 
calculation of the number of first- and second-degree co-arrestees who were homicide 
victims, inspired by prior work by Papachristos and colleagues (Papachristos 2009; 


Table 5 Timing of shooting associated with post-treatment arrest 


Group 

Total arrests for 
shooting 

Arrests with valid 
shooting date 

# Shootings after 
intervention 

% Shootings after 
intervention 

Treatment {n = 426) 

9 

5 

4 

80% 

Unweighted comparison 
group {n= 17,754) 

84 

68 

60 

88% 
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Papachristos et al. 2011, 2012). Although outside the scope of this study, there is clearly 
a question about how well the model performed in predicting homicide victimization. 
Those placed on the SSL were twice as likely to be victims of a homicide as others with 
arrest records, and 233 times more likely to be victims of homicide than the average 
Chicago resident. However, even with those increased odds, the individuals on SSL 
still only experienced a 0.7 % homicide rate over 12 months, illustrating how difficult it 
is to predict low-incidence events. Since the first list development, the algorithm has 
been improved, and now, according to ITT, 29 % of the top 400 subjects were 
accurately predicted to be involved in gun violence over an 18-month window 
(Lewin and Wemick 2015). 

In terms of how to operationalize the predictions, the algorithm did not differentiate 
between “high-risk” versus “high-threat” individuals, which was found to be problem¬ 
atic in the field. There were no formal differentiations in the SSL predictive model 
between persons who were “high threat” in terms of being violent threats to their local 
community and persons who were “high risk” of becoming a victim based on their 
associations or lifestyle attributes, for example, substance use disorders and gambling, 
but who were not violent themselves. Assuming these two groups require different 
interventions, group identification would be necessary information for police to devise 
targeted prevention strategies. According to IIT and CPD, in the newer iteration of the 
SSL, specific risk factors are presented along with the list of individuals to facilitate the 
appropriate selection of an intervention. 


Conclusions 

Although the findings of this study are more relevant for cities, particularly those 
experiencing gang-related homicides, the conclusions do offer important, general 
insights into the potential implications of predictive policing programs. The pilot effort 
does not appear to have been successful in reducing gun violence, although it may have 
improved justice by identifying more perpetrators. It does not appear that there were 
any unintended crime consequences, such as a violent backfire effect. The SSL 
program continues to evolve through improvements in the statistical algorithms of 
the prediction model. 

In terms of how the SSL program could be improved, it may be that both better 
predictions for the likelihood of being involved in gun violence and prevention 
strategies are necessary in order for an individual-based predictive policing program 
to work in the field. The discrepancy between observed outcomes and predicted risk is 
operationally significant, but statistically reasonably small given the difficulty of 
predicting a low-probability event. Also, it may be that the absolute quantification of 
the risk factor is less important in practice than the relative ranking of the subjects, 
which we did not attempt to measure. Since this study, based on version 1.0 of the 
model, the researchers who developed the SSL algorithm have progressively improved 
the predictive performance of the model, which is currently in version 4 (Lewin and 
Wemick 2015). 

Regardless of how “well” a model performs, there will always be a concern with 
misidentifying people as not going to commit or be a victim of gun violence (false 
negatives) and misidentification of people as going to commit or be a victim of gun 
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violence (false positives). This problem is well researched and the problem with false 
negatives and positives must be considered in terms of what it is being used for 
(Aitchison and Dunsmore 1980; Benjamini and Hochberg 1995). Models can be 
specified to penalize false negatives or positives more strongly, and it could be that a 
criminal justice application would have a different threshold than, for instance, a 
medical screening tool (Berk 2011). The problem is actually a fairly standard issue of 
modem statistical practice of large-scale multiple testing in the analysis of high¬ 
dimensional data. In their study to identify potentially problematic officers, for exam¬ 
ple, Ridgeway and MacDonald (2009) used a false discovery rate approach and 
selected a threshold of 0.50, which implies that the cost of failing to identify a problem 
officer equals the cost of flagging a good officer. Analysts and program developers 
should consider this trade-off for their purposes. Additionally, the consideration of this 
trade-off and thresholds should explicitly take into account the prevention strategy—if 
risk scores are used for a benign (and inexpensive) treatment, false positives are less of 
a problem; however, once the treatment becomes invasive, detrimental, and expensive, 
these false positives can become a huge problem. Conversely, false negatives in this 
application can be very costly, as they represent a missed opportunity to prevent 
shootings. 

Second, and perhaps more importantly, law enforcement needs better information 
about what to do with the predictions—^the “prevention” part of predictive policing. 
Indeed, the district commanders may have been cautious about intervening without the 
explicit direction of more senior administration, since these individuals were initially 
identified as potential victims. As this study shows, by providing almost no guidance to 
district commanders and police officers in the field, either up front or on an ongoing 
basis, they all generally opted for recommending to their officers to increase “contact” 
with individuals on the list in varying forms and levels of effort. And our analysis 
shows the officers did just that—^we find a statistically significant increase in police 
contacts. However, it is not at all evident that contacting people at greater risk of being 
involved in violence—especially without further guidance on what to say to them or 
otherwise how to follow up—is the relevant strategy to reduce violence. Alternatively, 
we did not find convincing evidence that increased contact resulted in a backfiring 
effect, but the possibility cannot be ruled out. 

The finding that the list had a direct effect on arrest, rather than victimization, raises 
privacy and civil rights considerations that must be carefully considered, especially for 
predictions that are targeted at vulnerable groups at high risk of victimization. Both 
local and national media openly ask whether the CPD SSL pilot constitutes racial 
profiling (Erbentraut 2014; Llenas 2014; Stroud 2014). A review of the legal and 
constitutional issues involved in using predictions for criminal justice purposes notes 
that, while it is not legal to use protected classes as predictors (Starr 2014), classifica¬ 
tions that have differential impact on different protected classes, such as racial groups, 
that are not designed to have this impact, are legal (Tonry 1987). Tonry (1987) also 
argues that the ethical issues with using prediction in a policing context are less 
controversial than in other criminal justice settings because they are necessary for the 
cost-effective distribution of scarce resources, and their decisions will ultimately be 
reviewed by impartial judges before punishment is delivered. However, using predic¬ 
tions to identify individuals in the commmunity for increased police scrutiny has not 
been subject to judicial review. 
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